feat(bench): add new benchmarking script, harness, and profiling guide #3840

abhiaagarwal · 2025-10-11T17:54:24Z

Description

This redoes the merge-based benchmark in crates/benchmark, replacing it with divan as a real harness combined with adding a script that can be used for profiling.

Related Issue(s)

Closes #3839

Documentation

Documentation is included in the updated README

github-actions · 2025-10-11T17:54:48Z

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

abhiaagarwal · 2025-10-11T17:55:46Z

Example profile (not very useful due to the nature of tokio/async, but wonder if there's a way around that): https://share.firefox.dev/4hcALa8

Benchmark results:

merge                                                                                 fastest       │ slowest       │ median        │ mean          │ samples │ iters
├─ delete_only                                                                                      │               │               │               │         │
│  ╰─ MergePerfParams { sample_matched_rows: 0.05, sample_not_matched_rows: 0.0 }     124.6 ms      │ 192.9 ms      │ 150.3 ms      │ 150.6 ms      │ 100     │ 100
│                                                                                     max alloc:    │               │               │               │         │
│                                                                                       19464       │ 19020         │ 19464         │ 19385         │         │
│                                                                                       34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
│                                                                                     alloc:        │               │               │               │         │
│                                                                                       398214      │ 396991        │ 398214        │ 398058        │         │
│                                                                                       76.32 MB    │ 76.21 MB      │ 76.32 MB      │ 76.31 MB      │         │
│                                                                                     dealloc:      │               │               │               │         │
│                                                                                       395903      │ 394704        │ 395886        │ 395718        │         │
│                                                                                       96.02 MB    │ 95.87 MB      │ 96.02 MB      │ 96.01 MB      │         │
│                                                                                     grow:         │               │               │               │         │
│                                                                                       17410       │ 17382         │ 17410         │ 17412         │         │
│                                                                                       14.38 MB    │ 14.34 MB      │ 14.37 MB      │ 14.39 MB      │         │
│                                                                                     shrink:       │               │               │               │         │
│                                                                                       85          │ 85            │ 85            │ 85.16         │         │
│                                                                                       2.777 KB    │ 2.777 KB      │ 2.777 KB      │ 8.515 KB      │         │
├─ multiple_insert_only                                                                             │               │               │               │         │
│  ├─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 0.05 }     65.7 ms       │ 111.6 ms      │ 69.46 ms      │ 69.91 ms      │ 100     │ 100
│  │                                                                                  max alloc:    │               │               │               │         │
│  │                                                                                    19175       │ 19621         │ 19175         │ 19179         │         │
│  │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
│  │                                                                                  alloc:        │               │               │               │         │
│  │                                                                                    420725      │ 425353        │ 420725        │ 420772        │         │
│  │                                                                                    78.13 MB    │ 78.6 MB       │ 78.13 MB      │ 78.14 MB      │         │
│  │                                                                                  dealloc:      │               │               │               │         │
│  │                                                                                    417681      │ 422618        │ 417678        │ 417733        │         │
│  │                                                                                    97.76 MB    │ 98.5 MB       │ 97.76 MB      │ 97.77 MB      │         │
│  │                                                                                  grow:         │               │               │               │         │
│  │                                                                                    17482       │ 17723         │ 17481         │ 17484         │         │
│  │                                                                                    14.39 MB    │ 14.62 MB      │ 14.39 MB      │ 14.4 MB       │         │
│  │                                                                                  shrink:       │               │               │               │         │
│  │                                                                                    157         │ 157           │ 157           │ 157           │         │
│  │                                                                                    3.929 KB    │ 3.929 KB      │ 3.929 KB      │ 3.929 KB      │         │
│  ├─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 0.5 }      95.64 ms      │ 187 ms        │ 101.9 ms      │ 105.4 ms      │ 100     │ 100
│  │                                                                                  max alloc:    │               │               │               │         │
│  │                                                                                    19175       │ 20077         │ 19175         │ 19203         │         │
│  │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
│  │                                                                                  alloc:        │               │               │               │         │
│  │                                                                                    420725      │ 422376        │ 420725        │ 421010        │         │
│  │                                                                                    78.13 MB    │ 78.4 MB       │ 78.13 MB      │ 78.16 MB      │         │
│  │                                                                                  dealloc:      │               │               │               │         │
│  │                                                                                    417711      │ 419864        │ 417703        │ 418018        │         │
│  │                                                                                    97.77 MB    │ 98.1 MB       │ 97.77 MB      │ 97.84 MB      │         │
│  │                                                                                  grow:         │               │               │               │         │
│  │                                                                                    17482       │ 17609         │ 17481         │ 17496         │         │
│  │                                                                                    14.39 MB    │ 14.41 MB      │ 14.39 MB      │ 14.41 MB      │         │
│  │                                                                                  shrink:       │               │               │               │         │
│  │                                                                                    157         │ 157           │ 157           │ 157           │         │
│  │                                                                                    3.929 KB    │ 3.929 KB      │ 3.929 KB      │ 3.929 KB      │         │
│  ╰─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 1.0 }      124.9 ms      │ 252.2 ms      │ 135.1 ms      │ 142.5 ms      │ 100     │ 100
│                                                                                     max alloc:    │               │               │               │         │
│                                                                                       18344       │ 20224         │ 19175         │ 19253         │         │
│                                                                                       34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
│                                                                                     alloc:        │               │               │               │         │
│                                                                                       423366      │ 422551        │ 420725        │ 421090        │         │
│                                                                                       78.4 MB     │ 78.43 MB      │ 78.13 MB      │ 78.18 MB      │         │
│                                                                                     dealloc:      │               │               │               │         │
│                                                                                       420690      │ 420117        │ 417734        │ 418141        │         │
│                                                                                       100.6 MB    │ 98.14 MB      │ 97.77 MB      │ 97.85 MB      │         │
│                                                                                     grow:         │               │               │               │         │
│                                                                                       17676       │ 17611         │ 17481         │ 17503         │         │
│                                                                                       14.59 MB    │ 14.41 MB      │ 14.39 MB      │ 14.41 MB      │         │
│                                                                                     shrink:       │               │               │               │         │
│                                                                                       157         │ 157           │ 157           │ 157           │         │
│                                                                                       3.929 KB    │ 3.929 KB      │ 3.929 KB      │ 3.929 KB      │         │
╰─ upsert_file_matched                                                                              │               │               │               │         │
   ├─ MergePerfParams { sample_matched_rows: 0.1, sample_not_matched_rows: 0.0 }      142.5 ms      │ 180.3 ms      │ 155.9 ms      │ 157 ms        │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    19145       │ 19371         │ 19370         │ 19484         │         │
   │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    460665      │ 461282        │ 461281        │ 461594        │         │
   │                                                                                    82.45 MB    │ 82.51 MB      │ 82.51 MB      │ 82.54 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    458405      │ 458979        │ 459006        │ 459305        │         │
   │                                                                                    101.9 MB    │ 101.9 MB      │ 101.9 MB      │ 102 MB        │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    18174       │ 18188         │ 18188         │ 18195         │         │
   │                                                                                    14.13 MB    │ 14.15 MB      │ 14.15 MB      │ 14.16 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    226         │ 226           │ 226           │ 226           │         │
   │                                                                                    4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.01, sample_not_matched_rows: 0.1 }     138.4 ms      │ 190.7 ms      │ 163.2 ms      │ 162.6 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    19874       │ 19426         │ 19649         │ 19649         │         │
   │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    462510      │ 461281        │ 461896        │ 461896        │         │
   │                                                                                    82.62 MB    │ 82.51 MB      │ 82.56 MB      │ 82.56 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    460180      │ 458983        │ 459602        │ 459591        │         │
   │                                                                                    102.1 MB    │ 101.9 MB      │ 102 MB        │ 102 MB        │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    18217       │ 18187         │ 18203         │ 18203         │         │
   │                                                                                    14.18 MB    │ 14.15 MB      │ 14.16 MB      │ 14.17 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    226         │ 226           │ 226           │ 226           │         │
   │                                                                                    4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.1, sample_not_matched_rows: 0.01 }     126.3 ms      │ 193.2 ms      │ 143.8 ms      │ 146.1 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    19874       │ 19424         │ 19311         │ 19633         │         │
   │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    462511      │ 461281        │ 460973        │ 461852        │         │
   │                                                                                    82.62 MB    │ 82.51 MB      │ 82.48 MB      │ 82.56 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    460193      │ 459005        │ 458682        │ 459540        │         │
   │                                                                                    102.1 MB    │ 101.9 MB      │ 101.9 MB      │ 102 MB        │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    18218       │ 18188         │ 18181         │ 18202         │         │
   │                                                                                    14.18 MB    │ 14.15 MB      │ 14.14 MB      │ 14.16 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    226         │ 226           │ 226           │ 226           │         │
   │                                                                                    4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.001, sample_not_matched_rows: 0.001 }  129.4 ms      │ 188.2 ms      │ 146.5 ms      │ 146.6 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    19875       │ 19199         │ 19424         │ 19653         │         │
   │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    462511      │ 460666        │ 462009        │ 461922        │         │
   │                                                                                    82.62 MB    │ 82.45 MB      │ 82.56 MB      │ 82.57 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    460158      │ 458397        │ 459713        │ 459607        │         │
   │                                                                                    102.1 MB    │ 101.9 MB      │ 102 MB        │ 102 MB        │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    18218       │ 18174         │ 18218         │ 18204         │         │
   │                                                                                    14.18 MB    │ 14.13 MB      │ 14.17 MB      │ 14.17 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    226         │ 226           │ 226           │ 226           │         │
   │                                                                                    4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.5, sample_not_matched_rows: 0.001 }    135.4 ms      │ 226.3 ms      │ 148.9 ms      │ 151.3 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    19874       │ 19874         │ 19649         │ 19637         │         │
   │                                                                                    34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    462511      │ 462511        │ 461896        │ 461865        │         │
   │                                                                                    82.62 MB    │ 82.62 MB      │ 82.56 MB      │ 82.56 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    460164      │ 460173        │ 459583        │ 459549        │         │
   │                                                                                    102.1 MB    │ 102.1 MB      │ 102 MB        │ 102 MB        │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    18218       │ 18218         │ 18203         │ 18202         │         │
   │                                                                                    14.18 MB    │ 14.18 MB      │ 14.16 MB      │ 14.16 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    226         │ 226           │ 226           │ 226           │         │
   │                                                                                    4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │
   ╰─ MergePerfParams { sample_matched_rows: 0.99, sample_not_matched_rows: 0.001 }   111.1 ms      │ 144.7 ms      │ 123.4 ms      │ 124.2 ms      │ 100     │ 100
                                                                                      max alloc:    │               │               │               │         │
                                                                                        19649       │ 19424         │ 19874         │ 19642         │         │
                                                                                        34.47 MB    │ 34.47 MB      │ 34.47 MB      │ 34.47 MB      │         │
                                                                                      alloc:        │               │               │               │         │
                                                                                        461896      │ 461281        │ 462511        │ 461877        │         │
                                                                                        82.56 MB    │ 82.51 MB      │ 82.62 MB      │ 82.56 MB      │         │
                                                                                      dealloc:      │               │               │               │         │
                                                                                        459557      │ 458978        │ 460169        │ 459564        │         │
                                                                                        102 MB      │ 101.9 MB      │ 102.1 MB      │ 102 MB        │         │
                                                                                      grow:         │               │               │               │         │
                                                                                        18204       │ 18188         │ 18218         │ 18202         │         │
                                                                                        14.17 MB    │ 14.15 MB      │ 14.18 MB      │ 14.17 MB      │         │
                                                                                      shrink:       │               │               │               │         │
                                                                                        226         │ 226           │ 226           │ 226           │         │
                                                                                        4.873 KB    │ 4.873 KB      │ 4.873 KB      │ 4.873 KB      │         │

codecov · 2025-10-11T18:31:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.98%. Comparing base (4acc60b) to head (74f4110).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3840      +/-   ##
==========================================
- Coverage   73.99%   73.98%   -0.02%     
==========================================
  Files         148      148              
  Lines       38904    38882      -22     
  Branches    38904    38882      -22     
==========================================
- Hits        28788    28767      -21     
  Misses       8850     8850              
+ Partials     1266     1265       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

roeap

Awesome, I was unaware that duckdb could also generate tcp-ds data!

Just wondering if we still need benchmarks to be in a separate crate? I might look into this in a follow up.

roeap · 2025-10-11T18:40:48Z

It seems we got rid of some dependency requirements...

abhiaagarwal · 2025-10-11T18:43:26Z

Awesome, I was unaware that duckdb could also generate tcp-ds data!

Just wondering if we still need benchmarks to be in a separate crate? I might look into this in a follow up.

Me neither! Super fast on my M4 Pro.

And I don't think so, I left it just to be consistent. It is nice to have the benchmark script as a separate binary, though

abhiaagarwal · 2025-10-11T18:49:42Z

don't approve yet, want to improve the harness to make sure we're only benchmarking the merge (and not the setup)

abhiaagarwal · 2025-10-11T19:07:15Z

Nice, much better

merge                                                                                 fastest       │ slowest       │ median        │ mean          │ samples │ iters
├─ delete_only                                                                                      │               │               │               │         │
│  ╰─ MergePerfParams { sample_matched_rows: 0.05, sample_not_matched_rows: 0.0 }     104 ms        │ 153.3 ms      │ 127.4 ms      │ 127.8 ms      │ 100     │ 100
│                                                                                     max alloc:    │               │               │               │         │
│                                                                                       17893       │ 16993         │ 17330         │ 17343         │         │
│                                                                                       1.611 MB    │ 1.553 MB      │ 1.575 MB      │ 1.576 MB      │         │
│                                                                                     alloc:        │               │               │               │         │
│                                                                                       364627      │ 362167        │ 363089        │ 363114        │         │
│                                                                                       37.56 MB    │ 37.33 MB      │ 37.42 MB      │ 37.42 MB      │         │
│                                                                                     dealloc:      │               │               │               │         │
│                                                                                       364215      │ 361822        │ 362709        │ 362735        │         │
│                                                                                       41.77 MB    │ 41.47 MB      │ 41.58 MB      │ 41.58 MB      │         │
│                                                                                     grow:         │               │               │               │         │
│                                                                                       15341       │ 15282         │ 15304         │ 15304         │         │
│                                                                                       4.24 MB     │ 4.174 MB      │ 4.199 MB      │ 4.2 MB        │         │
│                                                                                     shrink:       │               │               │               │         │
│                                                                                       80          │ 80            │ 80            │ 80            │         │
│                                                                                       1.76 KB     │ 1.76 KB       │ 1.76 KB       │ 1.76 KB       │         │
├─ multiple_insert_only                                                                             │               │               │               │         │
│  ├─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 0.05 }     42.9 ms       │ 75.49 ms      │ 48.01 ms      │ 49.17 ms      │ 100     │ 100
│  │                                                                                  max alloc:    │               │               │               │         │
│  │                                                                                    17154       │ 17154         │ 17154         │ 17154         │         │
│  │                                                                                    1.544 MB    │ 1.544 MB      │ 1.544 MB      │ 1.544 MB      │         │
│  │                                                                                  alloc:        │               │               │               │         │
│  │                                                                                    385908      │ 385908        │ 385908        │ 385908        │         │
│  │                                                                                    39.26 MB    │ 39.26 MB      │ 39.26 MB      │ 39.26 MB      │         │
│  │                                                                                  dealloc:      │               │               │               │         │
│  │                                                                                    384828      │ 384828        │ 384828        │ 384828        │         │
│  │                                                                                    43.37 MB    │ 43.37 MB      │ 43.37 MB      │ 43.36 MB      │         │
│  │                                                                                  grow:         │               │               │               │         │
│  │                                                                                    15383       │ 15383         │ 15383         │ 15383         │         │
│  │                                                                                    4.227 MB    │ 4.227 MB      │ 4.227 MB      │ 4.227 MB      │         │
│  │                                                                                  shrink:       │               │               │               │         │
│  │                                                                                    152         │ 152           │ 152           │ 152           │         │
│  │                                                                                    2.912 KB    │ 2.912 KB      │ 2.912 KB      │ 2.912 KB      │         │
│  ├─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 0.5 }      73.14 ms      │ 183.7 ms      │ 85.6 ms       │ 96.09 ms      │ 100     │ 100
│  │                                                                                  max alloc:    │               │               │               │         │
│  │                                                                                    17154       │ 17151         │ 17154         │ 17158         │         │
│  │                                                                                    1.544 MB    │ 1.543 MB      │ 1.544 MB      │ 1.544 MB      │         │
│  │                                                                                  alloc:        │               │               │               │         │
│  │                                                                                    385908      │ 385908        │ 385908        │ 385957        │         │
│  │                                                                                    39.26 MB    │ 39.26 MB      │ 39.26 MB      │ 39.26 MB      │         │
│  │                                                                                  dealloc:      │               │               │               │         │
│  │                                                                                    384852      │ 384852        │ 384852        │ 384904        │         │
│  │                                                                                    43.37 MB    │ 43.37 MB      │ 43.37 MB      │ 43.37 MB      │         │
│  │                                                                                  grow:         │               │               │               │         │
│  │                                                                                    15383       │ 15383         │ 15383         │ 15385         │         │
│  │                                                                                    4.227 MB    │ 4.227 MB      │ 4.227 MB      │ 4.229 MB      │         │
│  │                                                                                  shrink:       │               │               │               │         │
│  │                                                                                    152         │ 152           │ 152           │ 152           │         │
│  │                                                                                    2.912 KB    │ 2.912 KB      │ 2.912 KB      │ 2.912 KB      │         │
│  ╰─ MergePerfParams { sample_matched_rows: 0.0, sample_not_matched_rows: 1.0 }      99.17 ms      │ 203.5 ms      │ 108.1 ms      │ 121.9 ms      │ 100     │ 100
│                                                                                     max alloc:    │               │               │               │         │
│                                                                                       16323       │ 18350         │ 17154         │ 17319         │         │
│                                                                                       1.523 MB    │ 1.646 MB      │ 1.544 MB      │ 1.559 MB      │         │
│                                                                                     alloc:        │               │               │               │         │
│                                                                                       388549      │ 387600        │ 385927        │ 386368        │         │
│                                                                                       39.53 MB    │ 39.57 MB      │ 39.26 MB      │ 39.32 MB      │         │
│                                                                                     dealloc:      │               │               │               │         │
│                                                                                       387834      │ 387171        │ 384887        │ 385431        │         │
│                                                                                       46.2 MB     │ 43.76 MB      │ 43.37 MB      │ 43.5 MB       │         │
│                                                                                     grow:         │               │               │               │         │
│                                                                                       15578       │ 15507         │ 15383         │ 15413         │         │
│                                                                                       4.423 MB    │ 4.242 MB      │ 4.227 MB      │ 4.24 MB       │         │
│                                                                                     shrink:       │               │               │               │         │
│                                                                                       152         │ 152           │ 152           │ 152           │         │
│                                                                                       2.912 KB    │ 2.912 KB      │ 2.912 KB      │ 2.912 KB      │         │
╰─ upsert_file_matched                                                                              │               │               │               │         │
   ├─ MergePerfParams { sample_matched_rows: 0.1, sample_not_matched_rows: 0.0 }      111.1 ms      │ 165 ms        │ 132.9 ms      │ 133.5 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    17349       │ 17574         │ 17461         │ 17472         │         │
   │                                                                                    1.584 MB    │ 1.601 MB      │ 1.593 MB      │ 1.593 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    426464      │ 427079        │ 426771        │ 426802        │         │
   │                                                                                    43.63 MB    │ 43.69 MB      │ 43.66 MB      │ 43.66 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    426115      │ 426717        │ 426416        │ 426451        │         │
   │                                                                                    47.58 MB    │ 47.66 MB      │ 47.62 MB      │ 47.62 MB      │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    16089       │ 16105         │ 16097         │ 16097         │         │
   │                                                                                    3.979 MB    │ 3.999 MB      │ 3.989 MB      │ 3.99 MB       │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    221         │ 221           │ 221           │ 221           │         │
   │                                                                                    3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.01, sample_not_matched_rows: 0.1 }     121.9 ms      │ 205.5 ms      │ 139 ms        │ 141.6 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    17853       │ 17628         │ 17628         │ 17646         │         │
   │                                                                                    1.616 MB    │ 1.602 MB      │ 1.6 MB        │ 1.602 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    427694      │ 427079        │ 427079        │ 427128        │         │
   │                                                                                    43.74 MB    │ 43.69 MB      │ 43.69 MB      │ 43.69 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    427307      │ 426705        │ 426712        │ 426760        │         │
   │                                                                                    47.74 MB    │ 47.66 MB      │ 47.66 MB      │ 47.67 MB      │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    16119       │ 16105         │ 16104         │ 16105         │         │
   │                                                                                    4.014 MB    │ 3.999 MB      │ 3.997 MB      │ 3.999 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    221         │ 221           │ 221           │ 221           │         │
   │                                                                                    3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.1, sample_not_matched_rows: 0.01 }     102.8 ms      │ 152.2 ms      │ 122.9 ms      │ 123.2 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    17628       │ 18078         │ 17515         │ 17580         │         │
   │                                                                                    1.602 MB    │ 1.629 MB      │ 1.594 MB      │ 1.598 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    427079      │ 428309        │ 426771        │ 426949        │         │
   │                                                                                    43.69 MB    │ 43.8 MB       │ 43.66 MB      │ 43.67 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    426688      │ 427893        │ 426411        │ 426578        │         │
   │                                                                                    47.66 MB    │ 47.81 MB      │ 47.62 MB      │ 47.64 MB      │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    16105       │ 16134         │ 16097         │ 16101         │         │
   │                                                                                    3.999 MB    │ 4.03 MB       │ 3.989 MB      │ 3.994 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    221         │ 221           │ 221           │ 221           │         │
   │                                                                                    3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.001, sample_not_matched_rows: 0.001 }  102.9 ms      │ 175.2 ms      │ 121 ms        │ 123.1 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    18078       │ 17628         │ 17628         │ 17614         │         │
   │                                                                                    1.629 MB    │ 1.602 MB      │ 1.6 MB        │ 1.6 MB        │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    428309      │ 427079        │ 427079        │ 427042        │         │
   │                                                                                    43.8 MB     │ 43.69 MB      │ 43.69 MB      │ 43.68 MB      │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    427908      │ 426689        │ 426704        │ 426670        │         │
   │                                                                                    47.81 MB    │ 47.66 MB      │ 47.66 MB      │ 47.66 MB      │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    16134       │ 16105         │ 16104         │ 16103         │         │
   │                                                                                    4.03 MB     │ 3.999 MB      │ 3.997 MB      │ 3.997 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    221         │ 221           │ 221           │ 221           │         │
   │                                                                                    3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │
   ├─ MergePerfParams { sample_matched_rows: 0.5, sample_not_matched_rows: 0.001 }    108 ms        │ 155.4 ms      │ 117.4 ms      │ 118.4 ms      │ 100     │ 100
   │                                                                                  max alloc:    │               │               │               │         │
   │                                                                                    17628       │ 17628         │ 17741         │ 17670         │         │
   │                                                                                    1.602 MB    │ 1.602 MB      │ 1.609 MB      │ 1.604 MB      │         │
   │                                                                                  alloc:        │               │               │               │         │
   │                                                                                    427079      │ 427079        │ 427386        │ 427195        │         │
   │                                                                                    43.69 MB    │ 43.69 MB      │ 43.71 MB      │ 43.7 MB       │         │
   │                                                                                  dealloc:      │               │               │               │         │
   │                                                                                    426704      │ 426687        │ 426989        │ 426820        │         │
   │                                                                                    47.66 MB    │ 47.66 MB      │ 47.7 MB       │ 47.68 MB      │         │
   │                                                                                  grow:         │               │               │               │         │
   │                                                                                    16105       │ 16105         │ 16112         │ 16107         │         │
   │                                                                                    3.999 MB    │ 3.999 MB      │ 4.007 MB      │ 4.001 MB      │         │
   │                                                                                  shrink:       │               │               │               │         │
   │                                                                                    221         │ 221           │ 221           │ 221           │         │
   │                                                                                    3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │
   ╰─ MergePerfParams { sample_matched_rows: 0.99, sample_not_matched_rows: 0.001 }   85.76 ms      │ 128.8 ms      │ 95.99 ms      │ 97.88 ms      │ 100     │ 100
                                                                                      max alloc:    │               │               │               │         │
                                                                                        17854       │ 17403         │ 17740         │ 17623         │         │
                                                                                        1.616 MB    │ 1.585 MB      │ 1.609 MB      │ 1.601 MB      │         │
                                                                                      alloc:        │               │               │               │         │
                                                                                        427694      │ 426464        │ 427386        │ 427066        │         │
                                                                                        43.74 MB    │ 43.63 MB      │ 43.71 MB      │ 43.69 MB      │         │
                                                                                      dealloc:      │               │               │               │         │
                                                                                        427288      │ 426116        │ 427012        │ 426693        │         │
                                                                                        47.74 MB    │ 47.58 MB      │ 47.7 MB       │ 47.66 MB      │         │
                                                                                      grow:         │               │               │               │         │
                                                                                        16119       │ 16089         │ 16112         │ 16104         │         │
                                                                                        4.014 MB    │ 3.979 MB      │ 4.007 MB      │ 3.997 MB      │         │
                                                                                      shrink:       │               │               │               │         │
                                                                                        221         │ 221           │ 221           │ 221           │         │
                                                                                        3.856 KB    │ 3.856 KB      │ 3.856 KB      │ 3.856 KB      │         │

ion-elgreco · 2025-10-11T22:18:51Z

crates/benchmarks/README.md


-You can generate the TPC-DS dataset yourself by downloading and compiling [the generator](https://www.tpc.org/tpc_documents_current_versions/current_specifications5.asp) 
-You may need to update the CFLAGS to include `-fcommon` to compile on newer versions of GCC.
+This will generate a folder called `tpcds_parquet` containing many parquet files. Place it at `crates/benchmarks/data/tpcds_parquet` (or set `TPCDS_PARQUET_DIR`). Credits to [Xuanwo's Blog](https://xuanwo.io/links/2025/02/duckdb-is-the-best-tpc-data-generator/).


https://datafusion.apache.org/blog/2025/04/10/fastest-tpch-generator/

This one is faster than duckdb extension

ion-elgreco

Lgtm! Would only suggest using https://github.com/clflushopt/tpchgen-rs for faster generation

abhiaagarwal · 2025-10-11T22:40:49Z

Lgtm! Would only suggest using https://github.com/clflushopt/tpchgen-rs for faster generation

That's TPC-H, not TPC-DS. Looks like TPC-DS is still in progress clflushopt/tpchgen-rs#51

ion-elgreco · 2025-10-12T18:20:43Z

Lgtm! Would only suggest using https://github.com/clflushopt/tpchgen-rs for faster generation

That's TPC-H, not TPC-DS. Looks like TPC-DS is still in progress clflushopt/tpchgen-rs#51

Ah didn't see that :)

abhiaagarwal · 2025-10-12T23:56:53Z

No worries! When the TPC-DS support gets upstreamed, we'll presumably get the ability to generate the data on-the-fly

Signed-off-by: Abhi Agarwal <[email protected]>

…setup Signed-off-by: Abhi Agarwal <[email protected]>

abhiaagarwal requested review from hntd187, roeap and rtyler as code owners October 11, 2025 17:54

github-actions bot added the binding/rust Issues for the Rust crate label Oct 11, 2025

abhiaagarwal changed the title ~~Add new benchmark script, harness, and profiling guide~~ feat(bench): add new benchmarking script, harness, and profiling guide Oct 11, 2025

roeap previously approved these changes Oct 11, 2025

View reviewed changes

abhiaagarwal dismissed roeap’s stale review via 025c1e6 October 11, 2025 18:43

ion-elgreco reviewed Oct 11, 2025

View reviewed changes

ion-elgreco requested changes Oct 11, 2025

View reviewed changes

ion-elgreco enabled auto-merge (squash) October 13, 2025 08:06

ion-elgreco approved these changes Oct 13, 2025

View reviewed changes

abhiaagarwal added 3 commits October 13, 2025 11:54

Add new benchmark script, harness, and include profiling information

a76ff56

Signed-off-by: Abhi Agarwal <[email protected]>

Remove unused imports

9900fd9

Signed-off-by: Abhi Agarwal <[email protected]>

Make sure divan is only benchmarking the merge operation and not the …

74f4110

…setup Signed-off-by: Abhi Agarwal <[email protected]>

ion-elgreco force-pushed the abhi/divan-allocations branch from 4281a86 to 74f4110 Compare October 13, 2025 09:54

ion-elgreco merged commit b351f77 into delta-io:main Oct 13, 2025
26 checks passed

Uh oh!

feat(bench): add new benchmarking script, harness, and profiling guide #3840

feat(bench): add new benchmarking script, harness, and profiling guide #3840

Uh oh!

Conversation

abhiaagarwal commented Oct 11, 2025

Description

Related Issue(s)

Documentation

Uh oh!

github-actions bot commented Oct 11, 2025

Uh oh!

abhiaagarwal commented Oct 11, 2025

Uh oh!

codecov bot commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

roeap left a comment

Choose a reason for hiding this comment

Uh oh!

roeap commented Oct 11, 2025

Uh oh!

abhiaagarwal commented Oct 11, 2025

Uh oh!

abhiaagarwal commented Oct 11, 2025

Uh oh!

abhiaagarwal commented Oct 11, 2025

Uh oh!

ion-elgreco Oct 11, 2025

Choose a reason for hiding this comment

Uh oh!

ion-elgreco left a comment

Choose a reason for hiding this comment

Uh oh!

abhiaagarwal commented Oct 11, 2025

Uh oh!

ion-elgreco commented Oct 12, 2025

Uh oh!

abhiaagarwal commented Oct 12, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Oct 11, 2025 •

edited

Loading